prisma_cloud: Add misconfiguration and vulnerability data streams for CDR workflow #16012

kcreddy · 2025-11-18T11:46:56Z

Proposed commit message

prisma_cloud: Add misconfiguration and vulnerability data streams for CDR

Add 2 new data streams for fetching misconfigurations and vulnerabilities 
from Prisma Cloud API.

- ECS mapping and transforms have been added to facilitate with the
  Elastic Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflow.
- Minimum version constraints are raised as per Elasticsearch PR[3].
- Added dashboards for new data streams.
- Updated markdown to links panel in all dashboards.
- Test sample logs in this PR are taken from live data of Prisma 
  Cloud (anonymised).

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management
[3] https://github.com/elastic/elasticsearch/pull/138218

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Both pipeline and system tests pass.

```
--- Test results for package: prisma_cloud - START ---
╭──────────────┬──────────────────┬───────────┬──────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM      │ TEST TYPE │ TEST NAME                                            │ RESULT │ TIME ELAPSED │
├──────────────┼──────────────────┼───────────┼──────────────────────────────────────────────────────┼────────┼──────────────┤
│ prisma_cloud │ misconfiguration │ pipeline  │ (ingest pipeline warnings test-misconfiguration.log) │ PASS   │     443.47ms │
│ prisma_cloud │ misconfiguration │ pipeline  │ test-misconfiguration.log                            │ PASS   │   59.59125ms │
│ prisma_cloud │ vulnerability    │ pipeline  │ (ingest pipeline warnings test-vulnerability.log)    │ PASS   │ 407.008666ms │
│ prisma_cloud │ vulnerability    │ pipeline  │ test-vulnerability.log                               │ PASS   │  65.164875ms │
╰──────────────┴──────────────────┴───────────┴──────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: prisma_cloud - END   ---
Done
```

Transforms install correctly (see screenshot below).
Transforms have no authorisation error (see screenshot below).

Related issues

Screenshots

Dashboards

elastic-vault-github-plugin-prod · 2025-11-24T06:12:50Z

🚀 Benchmarks report

Package `prisma_cloud` 👍(2) 💚(3) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`alert`	2824.86	2304.15	-520.71 (-18.43%)	💔
`host_profile`	3039.51	2336.45	-703.06 (-23.13%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-11-25T05:21:55Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/prisma_cloud/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

alexreal1314 · 2025-11-26T13:17:17Z

packages/prisma_cloud/data_stream/misconfiguration/elasticsearch/ingest_pipeline/default.yml

+      copy_from: prisma_cloud.misconfiguration.scanned_policy.name
+      ignore_empty_value: true
+  - set:
+      field: rule.uuid


@kcreddy @maxcold I checked and we are still querying by resource.id and rule.id instead of the unique rule.uuid fields to get the document to display in the misconfiguration flyout. Since docs produced by this integration are not containing such field flyouts are unable to open correctly.

i would suggest for now to use to copy the rule.uuid to rule.id field as currently doing it for consistency only (using uuid) is possible, but associated with risks and might affect our native integration.

Done. Added rule.id from rule.uuid.

@kcreddy thanks, could you update the env with the changes? I would like to test also the contextual flow/alerts flyout.

@maxcold @kcreddy another weird issue that I see is Resource Type column is empty. after doing some research turns out we render the resource.sub_type under this column. Is it intentional?

prisma:

wiz:

@alexreal1314, resource.name is populated for Misconfigurations and only missing in Vulnerabilities workflow, which was discussed in the sheet.
Also, we don't have a rule.description field in the API data. Since its not Must Have, we have omitted it.

@kcreddy yep i meant resouce.id, I see that its currently missing.

Regarding rendering sub_type as Resource Type I think it is intentional and in line with out native integration. The sub type is more specific usually (when type and sub_type are not the same ofc) but for users we want to present only one field as Type . So that's why from data point of view it looks a bit off, but in the end users might not need this type/sub_type complexity. In our native type is more of group thing. Like monitoring is type and sub type is azure-network-watchers-flow-log so seeing monitoring in data grid isn't that helpful

@kcreddy yep i meant resouce.id, I see that its currently missing.

@alexreal1314, I just looked into security_solution-* indices and resource.id is present in all of them. Could it be that there's a bug in the Findings page, that you are not able to see it there?

@kcreddy LGTM besides the resource.sub_type which is missing, but I guess its not a blocker.

efd6 · 2025-12-03T20:43:52Z

packages/prisma_cloud/_dev/deploy/docker/files/config.yml

+        headers:
+          Content-Type:
+            - "application/octet-stream"
+        body: '{{file "/sample_logs/prisma-cloud-vulnerability.csv.gz"}}'


Add final new line.

efd6 · 2025-12-03T20:45:48Z

packages/prisma_cloud/data_stream/misconfiguration/agent/stream/cel.yml.hbs

+            ),
+            "next": {
+              ?"access_token": has(body.nextPageToken) ? optional.of(state.next.access_token) : optional.none(),
+              ?"pageToken": has(body.nextPageToken) ? optional.of(body.nextPageToken) : optional.none(),


Suggested change

?"pageToken": has(body.nextPageToken) ? optional.of(body.nextPageToken) : optional.none(),

?"pageToken": body.?nextPageToken,

Done in 6f33f5d

efd6 · 2025-12-03T20:49:01Z

packages/prisma_cloud/data_stream/misconfiguration/agent/stream/cel.yml.hbs

+              []
+            ),
+            "next": {
+              ?"access_token": has(body.nextPageToken) ? optional.of(state.next.access_token) : optional.none(),


Just checking that this is right. We're keeping the next access token that we had if there is a next page token? I think this is something that needs a little explanatory note.

That is correct. I added a comment.

efd6 · 2025-12-03T20:52:36Z

packages/prisma_cloud/data_stream/misconfiguration/agent/stream/cel.yml.hbs

+  want_more: false
+redact:
+  fields:
+    - user


Do we need to redact the user?

efd6 · 2025-12-03T20:58:31Z

packages/prisma_cloud/data_stream/vulnerability/agent/stream/cel.yml.hbs

+  want_more: false
+redact:
+  fields:
+    - user


Do we need to redact this?

efd6 · 2025-12-03T21:14:27Z

packages/prisma_cloud/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+  - append:
+      field: tags
+      value: preserve_original_event
+      allow_duplicates: false


Add final new line.

efd6 · 2025-12-03T21:15:24Z

packages/prisma_cloud/data_stream/misconfiguration/elasticsearch/ingest_pipeline/default.yml

+  - append:
+      field: tags
+      value: preserve_original_event
+      allow_duplicates: false


Add final new line.

Done in 6f33f5d

efd6 · 2025-12-03T21:41:35Z

Can you add an explanation for the source of the new test inputs?

kcreddy · 2025-12-04T05:20:47Z

Can you add an explanation for the source of the new test inputs?

@efd6, do you mean sample logs? I got them from the live data of Prisma Cloud.

elasticmachine · 2025-12-04T05:49:43Z

💚 Build Succeeded

Buildkite Build
Commit: 6f33f5d

History

💚 Build #34844 succeeded aff37ce
💚 Build #34746 succeeded 8003cef
💚 Build #34720 succeeded 198427f
💔 Build #34715 failed 0d7cfcb
💚 Build #34683 succeeded c714b98
💚 Build #34621 succeeded d04c695

cc @kcreddy

efd6 · 2025-12-04T06:10:31Z

do you mean sample logs? I got them from the live data of Prisma Cloud.

Thanks, can you note that in the proposed commit message?

elastic-vault-github-plugin-prod · 2025-12-04T06:33:29Z

Package prisma_cloud - 4.0.0 containing this change is available at https://epr.elastic.co/package/prisma_cloud/4.0.0/

Add template

d474724

kcreddy mentioned this pull request Nov 18, 2025

Add prisma_cloud data streams to kibana_system role permissions elastic/elasticsearch#138218

Merged

kcreddy changed the title ~~Add template~~ prisma_cloud: Add misconfiguration and vulnerability data streams for CDR Nov 18, 2025

kcreddy self-assigned this Nov 18, 2025

andrewkroh added the Integration:prisma_cloud Palo Alto Prisma Cloud label Nov 18, 2025

This was referenced Nov 19, 2025

Palo Alto Prisma Cloud: Implement mappings for Cloud Security Workflows #15059

Closed

Palo Alto Prisma Cloud: Implement transform for Cloud Security Workflows #15060

Closed

kcreddy added 8 commits November 20, 2025 23:55

Add misconfiguration pipeline and tests

a637ccb

update misconfig title

c513554

update misconfig description

91a6c18

Add external system ID to ECS id fields

a27279a

vulnerability template

9ef6826

Fix pipeline tests

a876896

Add system tests for vulnerability

e80b948

Fix pipeline tests

574caff

kcreddy added 2 commits November 24, 2025 19:31

Fix changes for ingestion

1de3130

Add transforms

169af2d

kcreddy marked this pull request as ready for review November 25, 2025 05:21

kcreddy requested a review from a team as a code owner November 25, 2025 05:21

kcreddy added Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Category: CDR labels Nov 25, 2025

kcreddy marked this pull request as draft November 25, 2025 05:23

kcreddy changed the title ~~prisma_cloud: Add misconfiguration and vulnerability data streams for CDR~~ prisma_cloud: Add misconfiguration and vulnerability data streams for CDR workflow Nov 25, 2025

Update README

d04c695

kcreddy marked this pull request as ready for review November 25, 2025 11:11

andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. breaking change labels Nov 25, 2025

update version numbers

c714b98

alexreal1314 reviewed Nov 26, 2025

View reviewed changes

packages/prisma_cloud/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

alexreal1314 reviewed Nov 26, 2025

View reviewed changes

kcreddy added 6 commits November 27, 2025 11:49

Address review comment

3e92d1b

refactor

0d7cfcb

update tests

198427f

Add workflow subcategory labels for integration setup from Findings page

8003cef

Add/update dashboards

ade4051

update changelog to reflect links panel addition

aff37ce

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Dec 1, 2025

alexreal1314 approved these changes Dec 3, 2025

View reviewed changes

efd6 reviewed Dec 3, 2025

View reviewed changes

Address PR comments

6f33f5d

kcreddy requested a review from efd6 December 4, 2025 06:03

efd6 approved these changes Dec 4, 2025

View reviewed changes

kcreddy merged commit 8ec8f45 into elastic:main Dec 4, 2025
8 checks passed

	?"pageToken": has(body.nextPageToken) ? optional.of(body.nextPageToken) : optional.none(),
	?"pageToken": body.?nextPageToken,

prisma_cloud: Add misconfiguration and vulnerability data streams for CDR workflow #16012

prisma_cloud: Add misconfiguration and vulnerability data streams for CDR workflow #16012

Uh oh!

Conversation

kcreddy commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package prisma_cloud 👍(2) 💚(3) 💔(2)

Uh oh!

elasticmachine commented Nov 25, 2025

Uh oh!

Uh oh!

alexreal1314 Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Dec 3, 2025

Uh oh!

kcreddy commented Dec 4, 2025

Uh oh!

elasticmachine commented Dec 4, 2025

💚 Build Succeeded

History

Uh oh!

efd6 commented Dec 4, 2025

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

kcreddy commented Nov 18, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Nov 24, 2025 •

edited

Loading

Package `prisma_cloud` 👍(2) 💚(3) 💔(2)

alexreal1314 Nov 26, 2025 •

edited

Loading

kcreddy Nov 27, 2025 •

edited

Loading